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Abstract — The problem of finding good linear codes for joint 
source-channel coding (JSCC) is investigated in this paper. By 
the code-spectrum approach, it has been proved in the authors' 
previous paper that a good linear code for the authors' JSCC 
scheme is a code with a good joint spectrum, so the main 
task in this paper is to construct linear codes with good joint 
spectra. First, the code-spectrum approach is developed further 
to facilitate the calculation of spectra. Second, some general 
principles for constructing good linear codes are presented. 
Finally, we propose an explicit construction of linear codes with 
good joint spectra based on low density parity check (LDPC) 
codes and low density generator matrix (LDGM) codes. 

I. Introduction 

A lot of research on practical designs of lossless joint 
source-channel coding (JSCC) based on linear codes have 
been done for specific correlated sources and multiple-access 
channels (MACs), e.g., correlated sources over separated noisy 
channels (e.g., [1]), correlated sources over additive white 
Gaussian noise (AWGN) MACs (e.g., [2]), correlated sources 
over Rayleigh fading MACs (e.g., [3]). However, for trans- 
mission of arbitrary correlated sources over arbitrary MACs, 
it is still not clear how to construct an optimal lossless JSCC 
scheme. In [4], we proposed a lossless JSCC scheme based on 
linear codes for MACs, which proved to be optimal if good 
linear codes and good conditional probabilities are chosen. 
Figure [T] illustrates the mechanism of our scheme in detail. 
Using the code-spectrum approach established in [4], we found 
that a good linear code for our JSCC scheme is a code with a 
good joint spectrum. Hence, to design a lossless JSCC scheme 
in practice, a big problem is how to construct linear codes 
with good joint spectra. In this paper, we will investigate the 
problem in depth and give an explicit construction of linear 
codes with good joint spectra based on sparse matrices. 

In the sequel, symbols, real variables and deterministic 
mappings are denoted by lowercase letters. Sets and random 
elements are denoted by capital letters, and the empty set is 
denoted by 0. Alphabets are denoted by script capital letters. 
All logarithms are taken to the natural base e and denoted by 
In. The composition of the functions / and g is denoted by 
/ o g, where (/ o g){x) = f{g{x)). The indicator function is 
denoted by !{•}. The cardinality of a set A is denoted by \A\. 
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For any random elements F and G in a common measurable 
space, the equality F ^ G means that F and G have the same 
probability distribution. 

II. Basics of the Code-Spectrum Approach 

Before investigating the problem of constructing good linear 
codes, we first need to briefly introduce our "code-spectrum" 
approach established in [4], which may be regarded as a 
generalization of the weight-distribution approach (e.g., [5]). 

Let X and y be two finite (additive) abelian groups. We 
define a linear code as a homomorphism / : A"" y"\ i.e., 
a map satisfying 

where X" and y"^ denote the n-fold direct product of X and 
the m-fold direct product of y, respectively, and x" denotes 
any sequence xiX2 ■ ■ ■ Xn in A"". We also define the rate of a 
linear code / to be the ratio n/m, and denote it by 

Note that any permutation (or interleaver) (t„ on n letters 
can be regarded as an automorphism on A"". We denote by I]„ 
a uniformly distributed random permutation on n letters. We 
tacitly assume that different random permutations occurring in 
the same expression are independent. 

Next, we introduce the concept of types [6]. The type of 
a sequence in A"" is the empirical distribution Px^^ on X 
defined by 



P,„(a) 
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For a (probability) distribution P on X, the set of sequences 
of type P in A" is denoted by Tp{X). A distribution P on 
X is called a type of sequences in X" if Tp{X) ^ 0. We 
denote by P{X) the set of all distributions on X, and denote 
by VniX) the set of all possible types of sequences in X^. 

Now, we introduce the spectrum, the most important concept 
in the code-spectrum approach. The spectrum of a nonempty 
set A C A"" is the empirical distribution Sx{A) on P{X) 
defined by 



Sx{A){P)^ 



|{x" e =P}| 
1^1 



VP e P(A). 



Analogously, the joint spectrum of a nonempty set B C A" x 
is the empirical distribution Sxy{B) on V{X) x Viy) 
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Fig. 1. The proposed linear codes based lossless joint source-channel encoding scheme of each encoder for multiple-access channels 



defined by 

Sxy{B){P,Q) 



\{{x^,y"^)GB\P, 



P,Py. 



Q}\ 



\B\ 



for all P e V{X),Q e V{y). Furthermore, we define the 
marginal spectra Sx{B), Sy{B) as the marginal distributions 
of Sxy{B), that is, 

Sx{B){P)^ ^ Sxy{B){P,Q) 
Qev{y) 

Sy{B){Q)^ ^ Sxy{B){P,Q). 
Pev(x) 

Please note that the summation in the definition of Sx{B){P) 
(or Sy{B){Q)) is taken over an infinite set Viy) (or ViX)), 
but is actually over a finite set because 

Sxy{B){P,Q)=0 

for any (P, Q) satisfying P S V{X)\Vn{X) or Q G 
'P{y)\Pm{y)- We define the conditional spectra Sy^x{B), 
Sx\y{B) as the conditional distributions of Sxy{B), that is. 



Sy^x{B){Q\P) 



Sx\y{B){P\Q) ^ 



Sxy{B){P,Q) 
Sx{B){P) 

VP satisfying Sx{B){P) ^ 

Sxy{B){P,Q) 

Sy{B){Q) 

VQ satisfying Sy{B){Q) 7^ 0. 



Then naturally, for any given function / : A"" y^, 
we can define its joint spectrum Sxy{f), forward conditional 
spectrum Sy^xif), backward conditional spectrum Sx\y{f), 
and image spectrum Sy(f) as Sxy{rl{f)), Sy^xi^'^K.f)), 
Sx\y{^Kf))' ™d 53;(rl(/)), respectively, where rl(/) is the 
relation defined by {(x", /(x"))|a;" G A""}. In this case, the 
forward conditional spectrum is given by 



Sy\x{mQ\P) = 



Sxy{.f){P,Q) 



Sx{X^){P) ■ 

If / is a Unear code, we further define its kernel spectrum as 
5Ar(ker /), where ker/ = {a;"|/(a;") = 0"*}. In this case, we 
have 

Syif) = Sy(J{X-)) 

since / is a homomorphism according to the definition of 
linear codes. 

The above definitions can be easily extended to more gen- 
eral cases. For example, we may consider the joint spectrum 



Sxyz{C) of a set C C A"" X 3;™ X Z\ or consider the 
joint spectrum Sx^x^y^yAQ) of ^ function g : X^^ x A*^^ 

yr' xyp. 

A series of properties regarding the spectrum of codes were 
proved in [4]. Readers may refer to [4] for the details. Some 
results are listed below for easy reference. 

Proposition 2.1: For all P G P„(A) and P, e P«.(A,) 
(1 < i < m), we have 



Sx{X''){P) 



\X\^'' 



Sx,-x^i[[A,){Pu-- ■,Pm) ^llSxAA){Pi), 



i=l 



where 



n 



and Ai C A"/*' (1 < i < m). 

Proposition 2.2: For any given random function F : A"" 

y"^, we have 

Pr{P(x") = y"} = |3^|-'"a(P)(P,.,P,^) (1) 
for any x" G A"", G 3^™, where 

P^S„oPoE„ (2) 

and 

E[Sxy{F){P,Q)] 



a{F){P,Q) = 



SxyiX" xy"^)iP,Q) 
E[Sy\x{F){Q\P)] 



(3) 



syiy"'m ■ 

Proposition 2.3: For any given linear code / : A"" —>■ y"*, 
we have 



a(,/)(^o",p,") = i:vri{y™-o"}. 



(4) 



If both X and y are the Galois field Fg, we define a particular 
random Unear code F^^%, : ^ F™ by a;" A„xn • 
x", where .t" represents an n-dimensional column vector, and 
Amxn denotes a random matrix with m rows and n columns, 
each entry independently taking values in F, according to a 
uniform distribution on ¥q. (Note that for each realization of 
Amxn, we then obtain a corresponding realization of F^^^. 
Such a random construction has already been adopted in [7, 
Section 2.1], [8], etc.) Then we have 



(5) 



for all a;" e F^\{0"} and G F™, or equivalently 

a{Ff^%){P,Q) = l (6) 
for all P e 7'„(Fg)\{Po4 and Q G P™(F,). 

III. Some New Results about Code Spectra 

In order to evaluate the performance of a linear code, we 
need to calculate or estimate its spectrum. However, the results 
established in [4] are still not enough for this purpose. So in 
this section, we will present some new results to facilitate 
the calculation of spectra. All the proofs are easy and hence 
omitted here. 

First, we proved the following two propositions, which 
imply that any codes (or functions) may be regarded as 
conditional probabihty distributions. Such a viewpoint is very 
helpful when calculating the spectrum of a complex code 
consisting of many simple codes. 

Proposition 3.1: For any random function F : — » y"^ 
and any x" G A"", we have 

Pr{(FoS„)(x") G T^m = E\Sy\x{F){Q\P^.)\. (7) 
Proposition 3.2: For any two random functions F : X" — > 
and G-.y"" ^ Z\ and any O G V„{X), Q G Vi{Z), we 
have 

E[S2ixiGoj:^oF)iQ\0)] 
= ^ E[Sy\AFmO)]E[Sz\yiG){Q\P)]. 

Second, let us develop a generating function method for the 
calculations of spectra. 

For any set A C X^, we define the generating function 
G{A) of its spectrum to be 

g{A){u)^ J2 S;,{A){P){u-P)^ 

where m is a map from X lo C (the set of complex numbers) 
or u G C'*', and for any u, w G C"^ , we define 

{ru){a) = ru{a) W GC,ae X, 
(w'')(a) =u(a)"(") yaeX, 

aex 

Also note that P G Vn{X) C C^. Analogously, for any set 
B C X" X y™-, we define the generating function of its joint 
spectrum as 



Q{B){u,v)^ 

J2 Sxy{B){P,Q){u"PUv"^% 



where u G C"^, v G C-*^. This in particular defines g{f) = 
e?(rl(/)) for any function / : A"" ^ J^™. 

Based on the above definitions, we proved the following 
properties. 



Proposition 3.3: For any two sets Ai C X"'^ and A2 C 
we have 

g{Ai X A2){u) = g{Ax){u) ■ g{A2){u). (8) 

For any two sets Ai C X" and A2 C 3^™, we have 

g{Ai X ^2)^,^;) = g{Ai){u) ■ g{A2){v). (9) 

For any two sets Bi C x y""^ and B2 C x y""^, 
we have 



g{Bi X B2){u, v) = g{Bi){u, v) • g{B2){u, v). 
Corollary 3.1: 

'(w)g 



(10) 



g{x-){u) 



\x\ 



where 

(")© ^ X! 

aex 

Corollary 3.2: For any two functions /i : <Y"i 
/2 : ^ we have 



y""^ and 



Gifi .f2){u,v) = g{h){u,v) • g{f2){u,v), (11) 

where /i © /2 is the map from to y^^+^^ defined 

by 

(/l © /2)(a;"^+"=) = fl{xi...m)f2{x^m+l)-im+n,)) 

for all a;"i+"2 g 

IV. General Principles for Constructing Linear 
Codes with Good Joint Spectra 

In this section, we will investigate some general problems 
for constructing Unear codes with good joint spectra. 

At first, we need to introduce some concepts of good 
linear codes. According to [4, Table I], a sequence of 5- 
asymptotically good (random) Unear codes F„ : A" — > 
for JSCC is a sequence of Unear codes whose joint spectra 
satisfy 



lim sup 



max 



1 E[Sxy{Fr,){P,Q)] ^ ^ 



PeVn{x)\{Pon}, n Sxy{X^ x yr^^){P, Q) 

And for comparison, a sequence of 5-asymptotically good 
(random) linear codes Fn ■ X^ y^n fQj- chaimel coding is 
a sequence of linear codes whose image spectra satisfy 

lim sup max In — ^ , ^ — — - — - < 0. 

When 6 equals zero, the above codes are then called asymp- 
totically good linear codes for JSCC and channel coding, 

respectively. 

From the Unearity of codes, it follows easily that 5- 
asymptotically good linear codes for JSCC are subsets of 
(5i?(F)-asymptotically good linear codes for channel coding, 
where R{F) = limsup„^^ i?(F„) and F = {F^j^^i. Then 
naturally, our first problem is: if a sequence of asymptotically 
good linear codes /„ for channel coding is given, can we find 
a sequence of asymptotically good linear codes <?„ for JSCC 



such that (7„(<Y") = fn{X")7 In other words (assuming that 
X = y = ¥q), if a sequence of asymptotically good channel 
codes is given, can we choose a good sequence of generator 
matrices so that the linear codes are asymptotically good for 
JSCC? 

When X = ¥g, the answer is positive, as a consequence of 
the following theorem. 



Theorem 4.1: For any linear code / : F^' 
exists a linear code g : F" ^ such that 



3^", there 



5(F") = /(F«) 



and 



Sy\^^{9){Q\P) < 



Sy{f{¥mQ) 



(12) 



for all P e 7'„(F,)\{Po4, Q £ V^{y). 

Proof: [Sketch of Proof] The main idea of the proof 
is to construct a random linear code G = / o F^-^*^^ = 
/ o !]„ o F^j^'^j, wher e Ff ^'^^ is defined in Proposition 123] 
Then by Proposition we have £;[53;|f,(G)((3|P)] = 
5j;(/(F^))(Q) for all F ^ Fo- and Q. This together with 
Proposition 14.11 (see below) then concludes the theorem. □ 
Proposition 4.1: Let rank(F) be the rank of the generator 



matrix of the linear code F : F^' 



F™. Then we have 



Pr{rank(Fj^L™)=m} = n(l 



i=l 



(13) 



where F^}f^ is defined in Proposition 12.31 and m < n. 
Furthermore, we have 

k 



n ' 



> 1 



q 



m—n—k 



q-l 



(14) 



i=l \ J / \ ^ / i=i 

where 1 < k < m. Let m = n and /c = 1, then we have 

Pr{rank(F«^,';) = n} > 1 - - q-\ (15) 
Proof: The identity ( fT3] l is a well known result in 
probability theory. To obtain a lower bound of the right hand 
side of ( fT3] l. we have 



m — k 



n 1 



> 1 



J-1 



m — k 

i=i 



= 1- 



n ' 

i=i ^ 

m 



> 1 



^711 — n — k \ ^ / 

Vr)nO 



This concludes (fT4l l. and ( fTSl l follows clearly. □ 
The above result does give a possible way for constructing 
good linear codes for JSCC based on good channel codes. 
However, such a construction is somewhat difficult to im- 
plement in practice, because the random generator matrix of 
^ij^n m is dense. Thus, our second problem is how to construct 



linear codes with good joint spectra based on sparse matrices 
so that known iterative encoding and decoding procedures 
have low complexity. The following theorem gives one feasible 
solution. 

Theorem 4.2: For a given sequence of sets {An C 
■Pm„ (A')\{Po'"" if there exist two sequences of random 

linear codes F„ : A""" and G„ : A"™" A"'" 

satisfying 

F„(A'"\{0"}) C y T'p^-{X) (16) 



and 



lim sup max 



In „ , . ., , < (17) 



respectively, where 6 > 0, then we have 

lim sup max 

„^oo OeV„(X)\{Po,^},QeVi„(X) 

1 E[Sxix{GnO^,n^oFn){Q\0)] 

Sxix^'^KQ) 

Proof: For all O e P„(A')\{Po"} and Q £ Pi,XX), and 
for any e > and sufficiently large n, we have 

oF„)(Q|0) 

- E E[S;,^;,{Gr.){Q\P)]E[S;,^;,{Fn){P\0)] 

Pev,„^(x) 

(b) 



J2 E[Sxix{Gn){Q\P)]E[Sxix{Fn){P\0)] 



(c) 



< J2 e-^'+^^Sx{X'-){Q)E[Sxix{F,,)iP\0)] 
PeAr, 

< e"^'+'^SxiX'")iQ), 



where (a) follows from Proposition 13. 21 (b) from the condition 
(fTSI l. and (c) from the condition ( fTTI l. Therefore, for any e > 
and sufficiently large n. 



max 

oev„{x)\{Po,^},QeVl^{x) 
E\Sx\x{Gn o 



im. 



oFn){Q\0)] 



Sx{X^"){Q) 



which establishes the theorem. □ 
Using Theorem l4.2l we can now construct good linear codes 
by a serial concatenation scheme, where the inner code is 
approximately 5-asymptotically good (satisfying (fTTT i) and the 
outer code is a linear code having good distance properties if 
we set An = {P G Vrn„iX)\l - P{0) > 7} in the condition 
( fT6] l. According to [9, Section IV], there exists a good low 
density parity check (LDPC) code F„ satisfying (fT6] l for an 
appropriate 7. Then om final problem is how to find a sequence 
of approximately 5-asymptotically good linear codes satisfying 
dnli with An = {P G 7',„„(A')|1 - F(0) > 7}. In the next 
section, we will find such codes in a family of codes called 
low density generator matrix (LDGM) codes. 



V. The spectra of LDGM Codes 

In this section, we will investigate the joint spectra of 
LDGM codes. We assume that the alphabet of codes is F^, 
and we denote a regular LDGM code by the map F^^^ : 
pi _^ ]pm defined by 



F, 



LD ^ /^m pCHK 



n,c,d 



(0™ l^r") ° Sen O (0r=i/f 



where nc = md, and is a single symbol repetition code 

ff^^ : ^ F= defined by 

f^'^^ (x) ^ XX ■ ■ ■ X \/xe¥g, 

and QYLiF^^^ denotes a parallel concatenation of m in- 
dependent copies of the random single symbol check code 
^CHK : -> defined by 



where C, (i = l,2,- -,c?) denotes an independent uniform 
random variable on the set Fg\{0}. 

To evaluate the joint spectrum or conditional spectrum of 
F^^ d' fii^st need to calculate the joint spectrum or condi- 
tional spectrum of /^^^ and F^^^. We have the following 
results. 

Proposition 5.1: 

gif^^^)iu^v)^-J2uiaMa)r, (18) 



a6F„ 



1 X ^ / n 



g" \nP 

PG-P„(F, 



(19) 



Sw,wM=jf^''){P, Q) = ^F,(F';)(P)1{P - Q}, (20) 



5F,|F,(0r=l/c )(QI^) = l{i^ = Q}■ 



(21) 



Proof: The identity (fTSl l holds clearly. From ( fTsT i and 
Corollary 13.21 we then have 



^?(0r=i/f^^)(«,«) 

i u{a)[v{a)r 



aew„ 



n 



Pe7'„(F,) ^ ^ aGF, 



- E 

PeP„(F,) ^ ^ 

This proves ( fT9] l. The identities ( |20] i and ( 1211 1 are easy conse- 
quences of ( fT9] l. □ 
In order to obtain the joint spectrum of F^^^, we need the 
following proposition (also well known), which can be easily 
proved by mathematical induction. 



Proposition 5.2: Let 



(22) 



where (i = 1, 2, cT) is an independent uniform random 
variable on the set Fq\{0}. Then we have 



Pv{Yd = a} = l{a = 0}- 

q 

l{a^O}- 

q 



i-(- 



1 sd-l 



q-1' 
q-l> , 



(23) 



for any a G GF(q). 

Now let us calculate the joint spectrum of F^^^. By Propo- 
sition 12.21 15.21 and Corollary 13.21 we obtained the following 
proposition. Its proof is long and hence omitted here, and 
readers may refer to [10] for the details. 

Proposition 5.3: 



/ qu{0) - (M)e 
I 9-1 



r,d+l 



(qviO) - (vU) 



(24) 



(25) 

i?[^F,F,(0™iF,c™)(P,Q)] < 92(0, P,Q), 

WO G Vrndi^g) (0(a) > 0,Va e {a|P(a) > 0}),(26) 

him\ 



- In a(0:^iFi^™)(P, Q) < 6d{P{0), Q(0)) + O 

\ III 

(21) 

where coef(/(M), (u")®) denotes the coefficient of {u")^ in 
the polynomial f{u), and 



.91 

A \mQl 



q' 



^{d+l) 



iiuur+iq-i)('-^^^^^^ 

\ 9-1 



mQ(0) 



(We)" 



92(0, P,Q) 

\rnQj 



qm{d+l) ^QmdP'^ 



qujO) - {u)q 
9-1 



1 + (9 - 1 



m(l-Q(0)) 



/ 9O(0)-l 
V 9-1 



iQ{0) 



m(l-Q(0)) 



Sdix,y) ^ inf { dD{x\\x) + yln 

0<S<1 



l + (9-l) 



qx — 1 
9-1 



+ (1 - y) In 



1 



f qx — 1 



V 9-1 

where /^(xlli) is the information divergence defined by 

D{x\\x) ^ j: In — + (1 — x) In r- 

X 1 ~ X 



(28) 



Based on the above preparations, we now start to analyze 



the joint spectrum of the regular LDGM code ^. 
Theorem 5.1: 
1 



ln«(F„^,^_,)(P,Q) < -<54P(0),Q(0)) + 



Inn 



where 5d is defined by ( |28] l. Let 

A„(7)^{PeP„(F,)|l-P(0)>7}, 



(29) 



(30) 



where < 7 < 1. Then, when q > 2, for any < 7 < 1 and 
any 5 > 0, there exits a positive integer c?o = '^o(7j'5) such 
that 

limsup max -\sia{F};^^ .)(P,Q) < 5 (31) 

for all integers d > do. 

Proof: At first, according to the definition of regular 
LDGM codes, we have 

E[Sr^wAF^ld)iQ\P)] 
'•^ J2 i?[^F,|F,(0r=i/f^^)(O|P)]- 

06P„c(F,) 

J2 HP = O}£;[5F,|r,(0lliF,™^)(Q|O)] 

OGP„e(F,) 
= i?[5F,|F,(0r4l^^r^)(Q|P)] 



where (a) follows from Proposition 13.21 and the definition of 
Prfcd^ (b) from Proposition 15. II and (c) from Proposition [ 
This then concludes (|29] |. 

By the definition of 5d, we have 



Sdix, y) < dD{x\\x) +y\n 
+ (1 - y ) In 



< In 



i + iqy- 1) 



l + (g-l^ 



qx — 1 

qx — 1 
Q-1 



qx ~ 1 



< (qy - 1) 



qx — I 



1 



<('7-l) 



qx — 1 



1 



Furthermore, when < a; < 1 — 7 and (7 > 2, we have 



'1 < 



1 qx — 1 

< < 1 



q-1- q-1 



or 



qx — 1 



q-i 



< max{ 



q-i 



,1- 



_ qi 

' q-l 

qi 



< 1 



q-i 



}<i. 



Then there exists a positive integer do = c?o(7, S) such that 

dS 

sup 5d{x,y) < — \fd>do. 

0<a;<l-7, C 
0<!/<l 



Note here that the ratio d/c is the rate of the code and hence 
should be a constant or at least bounded. 
Therefore, for any d > do, we have 

max - In a J (P,Q) 

i^) c flan 
< - sup Sd[x,y) + 

" 0<a;<l-7, \ n 

0<y<l 



< S+0 



Inn 



where (a) follows from ( |29] ). This concludes ( [3T1 ). □ 
Theorem 15.11 actually exhibits a family of codes whose 
joint spectra are approximately 5-asymptotically good. Then 
together with the conclusion at the end of Section HVl we have 
completed the construction of linear codes with good joint 
spectra, i.e., a serial concatenation scheme with one LDPC 
code as an outer code and one LDGM code as an inner code. 
An analogous construction has been proposed by Hsu in his 
thesis [11], but his purpose was only to find good channel 
codes and only a rate-1 LDGM code was employed as an 
inner code in his construction. 
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